A Whole Sentence Maximum Entropy Language Model

We introduce a new kind of language model, which models whole sentences or utterances directly using the Maximum Entropy paradigm. The new model is conceptually simpler, and more naturally suited to modeling whole-sentence phenomena, than the conditional ME models proposed to date. By avoiding the chain rule, the model treats each sentence or utterance as a \bag of features", where features are arbitrary computable properties of the sentence. The model is unnor-malizable, but this does not interfere with training (done via sampling) or with use. Using the model is computationally straightforward. The main computational cost of training the model is in generating sample sentences from a Gibbs distribution. Interestingly, this cost has different dependencies, and is potentially lower, than in the comparable conditional ME model. 1 Motivation Conventional statistical language models estimate the probability of an sentence s by using the chain rule to decompose it into a product of conditional probabilities: Pr(s) def = Pr(w 1 : : :w n) = n Y i=1 Pr(w i jw 1 : : :w i?1) def = n Y i=1 Pr(w i jh) (1) where h def = fw 1 ; : : :; w i?1 g is the history when predicting word w i. The vast majority of work in statistical language modeling is then devoted to estimating terms of the form Pr(wjh). The application of the chain rule is technically harmless since it uses an exact equality, not an approximation. However, terms like Pr(wjh) may not be the best way to think about estimating Pr(s): 1. Global sentence information such as grammaticality is awkward to encode in a conditional framework.